[FLINK-40300][python] Fix literal expression in Python Table API - #28924
[FLINK-40300][python] Fix literal expression in Python Table API#28924auroflow wants to merge 5 commits into
Conversation
auroflow
changed the title
auroflow
changed the title
Normalize explicitly typed literal values across Py4J before creating Table API expressions. Generated-by: OpenAI Codex (GPT-5)
Build reusable type-directed converter functions for nested literal values and document the Py4J boundary. Generated-by: OpenAI Codex (GPT-5)
Convert multiset elements recursively on both sides of the Py4J boundary and align the new Java code with project style. Generated-by: OpenAI Codex (GPT-5)
Convert temporal and array values before passing inferred literals through Py4J, while preserving Python array typecodes for Java type inference. Generated-by: OpenAI Codex (GPT-5)
1c23797 to
160530a
Compare
Route Table and DataFrame literals through one Java bridge that preserves Java type inference, validates nested values, and builds plannable composite expressions. Add planning and execution coverage for inferred, explicit, composite, and invalid literals. Generated-by: OpenAI Codex (GPT-5)
160530a to
7256a9c
Compare
auroflow
changed the title
| gateway = get_gateway() | ||
| jvm = gateway.jvm | ||
| if isinstance(value, datetime.datetime): | ||
| return _to_java_typed_literal_value(value, TimestampType()) |
There was a problem hiding this comment.
Could we preserve the offset/instant when inferring a literal from a timezone-aware datetime? This branch currently converts every datetime to LocalDateTime, so 2026-08-03 12:00+08:00 and 2026-08-03 04:00+00:00, which represent the same instant, become two different TIMESTAMP literals.
| elif isinstance(data_type, LocalZonedTimestampType) and isinstance( | ||
| value, datetime.datetime | ||
| ): | ||
| seconds = ( |
There was a problem hiding this comment.
This combines UTC-normalized whole seconds from utctimetuple() with the original local microsecond value.
For valid time-zone offsets containing fractional seconds, this creates the wrong instant. For example, 1970-01-01 00:00:00.100000+00:00:00.500000 should represent -0.4s, but the current code produces -0.9s.
Please normalize the complete datetime to UTC before extracting both seconds and nanoseconds. This would also allow awareness to be checked via utcoffset() and avoid losing fold information through timetuple()/mktime().
| return _unary_op("lit", v) | ||
| else: | ||
| return _binary_op("lit", v, _to_java_data_type(data_type)) | ||
| _j_literal_value = _to_java_literal_value(v, data_type) |
There was a problem hiding this comment.
Could we validate or convert data_type before passing it to _to_java_literal_value()?
With this current implementation, lit(1, "INT") now raises AttributeError: 'str' object has no attribute '_conversion_cls', whereas the previous implementation reported an appropriate TypeError from _to_java_data_type(). An explicit isinstance(data_type, DataType) check, or calling _to_java_data_type() first, would preserve the expected error behavior.
3 participants
What is the purpose of the change
Creating a PyFlink literal with
lit(value)andlit(value, data_type)requires the value to first cross the Py4J boundary and then pass Flink's Java-side literal validation. The current implementation can fail at either stage:datetime.date,datetime.time,datetime.datetime, anddatetime.timedeltaare not supported by the Py4J protocol directly. Therefore, literals declared asDATE,TIME,TIMESTAMP,TIMESTAMP_LTZ, or day-timeINTERVALfail before the Java Table API is invoked.Python numeric values are implicitly converted by Py4J into a limited set of Java boxed classes. A Python
intbecomesIntegerwhen it fits into 32 bits andLongotherwise, while a Pythonfloatalways becomesDouble. Consequently:TINYINTexpectsBytebut receivesInteger;SMALLINTexpectsShortbut receivesInteger;BIGINTexpectsLongbut receivesIntegerfor values within the 32-bit range;FLOATexpectsFloatbut receivesDouble.These values are rejected by
ValueLiteralExpression.validateValueDataType().Without an explicit data type, Python lists and tuples reach Java as generic containers. Java literal inference requires concrete Java arrays to infer the element type, particularly for empty
array.arrayvalues and nested arrays.The same issue affected nested values in arrays, maps, multisets, and rows.
This change converts Python-only values before transport, normalizes boxed numeric values in Java, and creates the literal in the same JVM call so that Py4J cannot convert the normalized value back to a Python primitive. The goal is to align the functionality of PyFlink Table API
litwith Java Table APIlit.Brief change log
ARRAY,MAP, andROWexpressions.Verifying this change
This change added tests and can be verified as follows:
PythonTableUtilsTestcoverage for Py4J numeric representations, year-month intervals, Java array inference, typed nulls, and invalid composite literals.Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: OpenAI Codex (GPT-5)